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ABSTRACT 



This paper identifies issues in defining and reporting 
performance standards for the 2002 National Assessment of Adult Literacy 
(NAAL) . The paper describes a continuum of conceptualizations of performance 
standards for adult literacy extending from "technical" conceptions of 
performance standards, as used in the psychometric literature, through 
"policy and programmatic" conceptions of adult literacy performance standards 
defined in educational goals, accountability systems, and in adult literacy 
program curricula, to "popular" conceptions of adult literacy performance 
standards as expressed in everyday speech and in the news media. The paper 
concludes that gaps between technical, policy and programmatic, and popular 
conceptions of adult literacy performance standards can be bridged by the 
following actions: (1) emphasizing applications of NAAL results to profile 

populations in need of literacy education services and avoiding 
misapplications of the results for adult education program accountability; 

(2) clarifying the differences between literacy constructs and skills 
measured by the NAAL and those measured by standardized tests used in 
accountability systems; and (3) making connections and clarifying differences 
between NAAL literacy definitions and performance levels and adult literacy 
program curricular content and content standards. (Contains 44 references.) 
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Abstract 



This paper identifies issues in defining and reporting performance standards for the 2002 National 
Assessment of Adult Literacy (NAAL). The paper describes a continuum of conceptualizations of 
performance standards for adult literacy extending from “technical” conceptions of performance standards, 
as used in the psychometric literature, to “policy/programmatic” conceptions of adult literacy performance 
standards, as defined in educational goals, accountability systems, and in adult literacy program curricula, 
to “popular” conceptions of adult literacy performance standards as expressed in everyday discourse and in 
the news media. The paper concludes that gaps between technical, policy/programmatic, and popular 
conceptions of adult literacy performance standards can be bridged by (1) emphasizing applications of 
NAAL results to profile populations in need of literacy education services and avoiding misapplications of 
the results for adult education program accountability; (2) clarifying the differences between literacy 
constructs and skills measured by the NAAL and those measured by standardized tests (CASAS, TABE, 
etc.) used in accountability systems; and, (3) making connections and clarifying differences between 
NAAL literacy definitions and performance levels and adult literacy program curricular content and content 
standards. 
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Introduction 



This paper addresses some of the issues raised by the following topic from the Education Statistics 

Services Institute Call for Papers for the Next National Assessment of Adult Literacy : 

Standard setting and benchmarking. Should there be performance standards 
established for the national assessment of adult literacy? What standard setting 
procedures are most appropriate for adult literacy and why? 

To clarify what is at stake in standard setting and benchmarking for the 2002 National Assessment 
of Adult Literacy (NAAL), the paper presents a simple typology of forms and functions for adult literacy 
performance standards. The term “performance standard” has been used to refer to a number of different 
(though related) things in discussions of educational research, policy, and practice. In its most generic 
sense, an adult literacy performance standard might be defined simply as an answer to the question: “how 
much literacy is enough.” At first glance it may seem that the most practical answer to the question of 
whether there should be performance standards established for the NAAL is no. The desire for comparable 
trend data would argue for replicating the “profile” approach to large-scale literacy assessment employed in 
the 1992 National Adult Literacy Survey (NALS) and related surveys. The NALS profile approach was not 
designed to answer the question of “how much literacy is enough.” Nonetheless, shortly after the release of 
the first NALS report, the National Education Goals Panel (NEGP) adopted the percentage of adults at or 
above Level 3 on the NALS prose scale as an indicator of progress toward Goal 6: Adult Literacy and 
Lifelong Learning, and thus created a de facto NALS-based performance standard for adult literacy. 

This paper looks at the NALS from the perspective of a variety of conceptualizations of 
performance standards for adult literacy. These conceptualizations span a range of issues in defining and 
reporting performance standards, from technical issues related to methods of setting cut scores, to policy 
and programmatic implications of literacy constructs and benchmarks, to public perceptions of the need for 
adult literacy educational services and of the quality of the existing system of adult basic education. Since 
the first NALS report was released in 1993 (Kirsch et al. 1993), there have been a number of pertinent 
developments in educational policy and large-scale assessment methodology. On the measurement side, 
psychometricians have proposed more broadly inclusive methods for setting performance standards. On the 
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policy side, a number of initiatives have moved the field of adult literacy education toward an assessment- 
driven, standards-based model of system reform and accountability. These developments present both new 
opportunities and new challenges for the definition and reporting of performance standards for the 2002 
NAAL. 

The NALS used state-of-the-art measurement methods and, in many respects, pushed the envelope 
of large-scale literacy assessment in profiling and reporting levels of literacy proficiency among adults in 
the United States. At the time that the NALS was designed and conducted, the American educational 
measurement and educational policy communities were witnessing the emergence of two intersecting 
trends. The first trend was the rise of large-scale performance assessment and the second was standards- 
based educational reform. These two trends have continued throughout the 1990s. By the time the NALS 
was first reported in 1993, the public (including policymakers and many practitioners) was beginning to 
express frustration with changes in assessment and reporting conventions. While innovations in large-scale 
assessment methodology were welcomed by researchers and assessment specialists, policymakers and the 
general public were having difficulty understanding and accepting as legitimate the results of large-scale 
performance assessments. Standards-based educational reform has fared better. Throughout the 1990s, the 
logic of an assessment-driven and standards-based system for educational reform and accountability has 
steadily gained broad-based acceptance. Continuing misunderstanding and mistrust of large-scale 
assessment results and heightened interest in educational program accountability measures will pose 
significant challenges for the NAAL. 

The analyses in this paper are based on a review of literature and interviews with stakeholders. In 
part, this paper is a follow-up and elaboration upon analyses contained in an earlier paper on issues in 
applying a standards-based educational reform model to the field of adult literacy (Stites, Foley, and 
Wagner 1 995). That paper noted the fragmented nature of the discourse of educational standards as it had 
been applied to the field of adult literacy. In the latter half of the 1990s, that fragmentation has continued 
and, to a degree, has been further complicated by, steadily increasing pressures for more rigorous systems 
of accountability for adult basic education programs. Reports of NALS results showing a high percentage 
of adult Americans at the lowest levels of literacy performance contributed to heightened demands for 
accountability and for improved practice and outcomes from adult basic literacy education. One response to 
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such demands has been additional impetus for the development of educational standards for adult literacy. 
Reflecting the general state of the adult basic education system, these standards-setting and accountability 
initiatives have themselves been somewhat fragmented and disjointed. Depending upon the approaches 
chosen in defining and reporting performance standards, the NAAL has the potential either to further 
fragment or to help create a more coherent system of adult literacy standards and accountability. 

This paper takes a critical look at the NALS approach to setting and reporting performance 
standards from technical, policy/programmatic, and finally, from popular perspectives. Within each 
perspective, selected aspects of the NALS approach to performance standards are briefly described and 
critiqued for the purpose of suggesting modifications that might be applied to the definition and reporting 
of performance standards for the NAAL. One general conclusion is that some of the gaps between 
technical, policy/programmatic, and popular conceptualizations of adult literacy performance standards can 
and should be bridged. At the end of this paper, a set of recommendations is made for defining and 
reporting performance standards for the next NAAL in ways that might make that bridging possible. 

Unpacking performance standards 

Because the term “performance standards” means different things in different contexts, the various 
ways that the phrase “adult literacy performance standards” might be interpreted require some clarification. 
This paper describes a continuum of conceptualizations of adult literacy performance standards. At one end 
of the continuum are various “technical” conceptions of performance standards as used in the psychometric 
literature (and in the NALS) in discussions of methods for setting cut scores for large-scale assessments. At 
the opposite end of the continuum are “popular” conceptions of performance standards as expressed in 
everyday discourse and in the news media. The “technical” end of the continuum is marked by 
empirical/theoretical conceptualizations of performance standards. The “popular” end of the continuum 
moves toward experiential/normative conceptualizations. In between these two extremes are 
“policy/programmatic” conceptions of adult literacy performance standards. Policy and programmatic 
performance standards blend varying degrees of empirical/theoretical and experiential/normative 
conceptualizations and are more directly linked to literacy educational practice than technical or popular 
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conceptualizations. Policy/programmatic performance standards are defined in educational goals and 
standards by such entities as the National Education Goals Panel (NEGP), the Secretary’s Commission on 
Achieving Necessary Skills (SCANS), the National Skills Standards Board (NSSB), and the National 
Institute for Literacy’s (NIFL) Equipped For the Future initiative, as well as in accountability systems, such 
as the Office of Vocational and Adult Education’s (OVAE) National Reporting System and state plans for 
adult education (mandated by the Workforce Investment Act of 1998). Examining the various forms and 
functions for adult literacy performance standards that have been defined at various points along this 
continuum as well as the potential connections among them will provide the background and rationale for a 
set of recommendations for defining and reporting performance standards for the planned 2002 NAAL. 

As noted above, the designers of the NALS did not intend for the survey to answer the question of 
“how much literacy is enough.” The authors of the first report of the NALS took some pains to steer 
readers away from seeing the NALS as setting this sort of simple performance standard. The Executive 
Summary of the 1993 NALS report included the following passage in a section entitled “Reflecting on the 
Results:” 



In reflecting on the results of the National Adult Literacy Survey, many readers will 
undoubtedly seek an answer to a fundamental question: Are the literacy skills of 
America’s adults adequate? That is, are the distributions of prose, document, and 
quantitative proficiency observed in this survey adequate to ensure individual 
opportunities for all adults, to increase worker productivity, or to strengthen America’s 
competitiveness around the world? 

Because it is impossible to say precisely what literacy skills are essential for individuals 
to succeed in this or any other society, the results of the National Adult Literacy Survey 
provide no firm answers to such questions. (Kirsch et al. 1993, xvii) 



The lack of intent to define a generic performance standard for adult literacy is further reinforced 
in the section of the report describing the work of the NALS Literacy Definition Committee. In a passage 
explaining the committee’s decision to adopt the functional definition of literacy as well as the three scales 
that were developed as part of what is called here the 1985 Young Adult Literacy Survey (YALS) (Kirsch 
and Jungeblut 1 986) and the 1989-90 U.S. Department of Labor survey of the literacy skills of job seekers 
(Kirsch and Jungeblut 1 992), the 1 993 report contrasts the NALS approach with “traditional approaches to 
literacy assessment” that distinguish “literates” from “illiterates.” 
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The literacy scales provide a useful way to organize a broad array of tasks and to report 
the assessment results. They represent a substantial improvement over traditional 
approaches to literacy assessment, which have tended to report on performance in terms 
of single tasks or to combine the results from diverse tasks into a single, conglomerate 
score. Such a score fosters the simplistic notion that “literates” and “illiterates” can be 
neatly distinguished from one another based on a single cut point on a single scale. The 
literacy scales, on the other hand, make it possible to profile the various types and levels 
of literacy among different subgroups in our society. In so doing, they help us to 
understand the diverse information-processing skills associated with the broad range of 
printed and written materials that adults read and their many purposes for reading them. 

In adopting the three scales for use in this survey, the committee’s aim was not to 
establish a single national standard for literacy. Rather, it was to provide an interpretive 
scheme that would enable levels of prose, document, and quantitative performance to be 
identified and allow descriptions of the knowledge and skills associated with each level 
to be developed. (Kirsch et al. 1993, 4) 



The brief passage quoted above contains a wealth of information about the NALS approach to 
large-scale literacy assessment. As the passage makes clear, the NALS design incorporated a conception of 
literacy proficiency as a complex performance rather than as a unidimensional skill. The NALS “profile” 
approach to literacy assessment described succinctly in the above passage also incorporated innovative 
Item Response Theory (IRT) methods for scaling tasks (test items) and individual responses to tasks for 
level of difficulty. Although, as stated in the passage, the intent of the NALS was not to set a “single 
national standard for literacy,” the passage may be interpreted as indicating several ways in which the ETS 
researchers who designed and conducted the NALS (and its predecessors, the YALS and the U.S. 
Department of Labor studies) approached the question of defining and reporting performance standards. 
Expressed in terms of the continuum of performance standards conceptualizations discussed above, these 
approaches may be stated as follows: 

(1 ) The constructs of literacy proficiency and IRT methods used in the NALS to design tasks and 
to establish (multiple) cut scores for performance levels constituted performance standard 
setting in the technical sense. 

(2) The adoption of a definition of literacy and design of tasks to reflect a theoretical construct of 
literacy proficiency as well as purposes for literacy constituted performance standard setting 
in the policy and programmatic senses. 

(3) The reporting of distributions of types and levels of literacy among the adult population 
represented performance standard setting in the popular sense. 
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The first of the three assertions above is likely to be uncontroversial. Because they were unintended and 
unforeseen in the design of the NALS, the second and the third assertions are more debatable. The 
unintended impact of the NALS on adult literacy performance standards in the policy and programmatic 
senses had much to do with the public policy climate (i.e., the push for educational standards and for 
standards-based accountability) at the time that NALS results were released. The unforeseen impact of the 
NALS on adult literacy performance standards in the popular sense was largely the result of widespread 
misinterpretation of the NALS scales and performance levels. The section below explores some issues that 
arise from the NALS approach to performance standards setting from a technical perspective with an eye to 
the possibilities for technical modifications of that approach for the NAAL. This is followed by a 
discussion of the NALS approach to performance standard setting and reporting from policy and 
programmatic perspectives again with an eye to planning for the NAAL. Finally, the impact of the NALS 
and the potential for the NAAL to shape popular perceptions of adult literacy performance standards are 
considered. 

Technical perspectives 

Within educational measurement circles, the term “performance standards” is most often used to 
refer to the cut scores used to mark levels of performance on a scale of skills and knowledge. Although 
there is no consensus on a method for setting performance standards in the psychometric literature, the 
traditional psychometric perspective generally views standards as “cut scores” (the numeric outcomes of a 
standard setting process), benchmarks on a scale, threshold values between contiguous categories, or 
numeric values that operationalize “‘how good is good enough’ (Livingston 1995, 39)” (Crocker and Zieky 
1 995, ES2). In the technical sense, the statement in the introduction (above) that the NALS was not 
intended to set a performance standard for adult literacy is false, though it remains true in the popular and 
policy/programmatic senses (see sections below). However, given the fact that the NALS levels were later 
interpreted as performance standards it would not be wrong to conclude that the NALS process for 
establishing cut scores for these levels constituted a technical approach to setting performance standards. 
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Even from the technical perspective, setting performance standards always involves a degree of 

subjectivity and arbitrary judgment. There seems to be a fair degree of unanimity among psycho metricians 

on the point that the answer to the question “how good is good enough” can only be answered by 

someone’s judgement. In an article published 1996, Ronald Berk counts “nearly 50 standards-setting 

methods documented in the literature” and goes on to point out: 

At the epicenter of every method proposed since prehistoric times is human judgment, 
whose subjectivity and imprecision wreak havoc in the minds of quantitatively trained 
people. (Berk 1996, 215) 

There also seems to be widespread agreement on the point that different methods for setting 
performance standards result in different standards (see Crocker and Zieky 1995; Jaeger et al. 1996). This 
is not the place, nor is it within this author’s competence to evaluate the IRT methods used in the NALS or 
to critique the general compatibility of IRT methods with stakeholder participation in standard setting. 
However, based on the evidence at hand, a case can be made that a process for expanding stakeholder 
participation in setting cut scores for the NAAL is both possible and desirable. The Item Response Theory 
(IRT) methods used for setting and validating cut scores for the NALS place greater weight on theoretical 
constructs and empirical analyses than on stakeholder judgments. Thus it is not surprising that participants 
in the stakeholder focus groups conducted by NCES in 1998 raised several concerns related to the setting of 
cut scores for the NALS. Among the most prominent of these concerns were that the range of literacy skills 
encompassed by Level 1 was too broad, that the RP80 (80 percent response probability) standard was too 
high, and that the NALS scales and levels were too difficult to understand and interpret (Sherman et al. 
1998). While it may not be possible or desirable to change either the range of skills in Level 1 or the RP80 
standard, it may be possible to adopt procedures for including stakeholders in technical performance 
standard setting processes for the NAAL in ways that would increase the “transparency” of the NALS 
scales and levels. 

Peter Mosenthal (1997) argues that the NALS approach for setting cutoff scores for performance 
levels was “a judgmental activity that was informed by an extensive empirical process” (297) and, citing 
Gary Phillips (1994), characterizes “this continual interaction between informed judgment and useful 
empirical data” as providing a basis to set “reasonable” standards. The following passage from the 1994 
article by Phillips gives a sense of what is meant by a “reasonable” standard setting process in this instance. 
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(Some) education performance standards require significantly more empirical data before 
a standard can be reasonably set. This is the case when educators attempt to establish 
acceptable levels of literacy for the adult population, or determining the level of 
proficiency needed in reading, writing and mathematics in order to be promoted from one 
grade level to another. ... These types of performance standards cannot be set solely on 
the basis of a political process. Although the standards are ultimately judgmental, they 
involve more than human intuition only. They require a consensus on what is meant by 
notions such as ‘literacy,’ ‘proficiency,* and ‘success.* Once these hypothetical 
constructs are defined, reliable, valid, and fair measures of them must be developed. ... 
The continual interaction between informed judgement and useful empirical data is all 
part of the art and science of setting performance standards in education. (Phillips 1994, 
192) 



As this passage makes clear, consensus on key constructs (literacy and proficiency in the case of the 
NALS) is the essential underpinning for reliable, valid, and fair measurement. Note that the form of 
consensus implied here is not a broad-based participatory process. Rather, the consensus building process 
implied is closer to the scientific method of hypothesis formation and testing. The literacy constructs 
underlying the NALS prose, document, and quantitative scales emerged from previous studies by ETS 
researchers, as noted in the first report. 

Previous research conducted at ETS has shown that the difficulty of a literacy task, and 
therefore its placement on a particular literacy scale, is determined by three factors: the 
structure or linguistic format of the material, the content and/or context from which it is 
selected, and the nature of the task, or what the individual is asked to do with the material 
(Kirsch et al. 1993, 69-70). 

As pointed out in the NAAL stakeholder focus groups (Sherman et al. 1 998, 5), the complexity of the 
resulting NALS scales and levels created difficulties in communicating the results to the public. 

The call for performance standards for adult literacy that are more “transparent” (easily 
understood) is likely to increase in the coming years. This will present an important challenge for the 
NAAL. As the trend toward more performance-based assessment continues, educators and assessment 
specialists are likely to join in the call for assessment designs that clearly communicate learning goals and 
processes (see policy/programmatic section below). 

Many psycho metricians have recently considered the challenges to standards-setting procedures 
posed by complex, performance-based assessment methods. In general, the response has been to call for 
new methods for including expert and stakeholder judgments of performance while preserving valid and 
reliable standards-setting processes. Berk (1996) identifies two changes in testing practices that have 
necessitated the development of new approaches to performance standards-setting: polytomous item 
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formats and multiple cut scores. Polytomous (as opposed to dichotomous) item formats have become 
common as large-scale assessments have incorporated constructed-response items (essays, oral discourse, 
portfolios and other formats that are scored using a rubric with a range of zero to two or greater). Likewise, 
use of multiple cut scores in large-scale assessments has been popularized by the National Assessment of 
Educational Progress (NAEP) as well as the NALS. In Berk’s view, the combination of these two changes 
in testing practices suggest a need for new judgmental standards-setting processes. He describes 10 
procedures for a General Eclectic Method (GEM) that he feels summarize the best of past practice as well 
as promising new techniques. Jaeger and colleagues (Jaeger et al. 1996) also see a need for new and more 
inclusive standard setting processes. They describe current methodology for setting performance standards 
on performance assessments as “embryonic” (80). While conceding that setting performance standards is “a 
judgmental process” and “subjective by definition” they argue for a performance standards-setting that is 
“arbitrary” (in a footnote they point to an Oxford English Dictionary definition of arbitrary as “relating to, 
or dependent on the discretion of the arbiter, arbitrator, or other legally-recognized authority; discretionary, 
not fixed”) but not “capricious” (also defined in a footnote as “Full of, subject to, or characterized by 
caprice; guided by whim or fancy rather than by judgment or settled purpose; whimsical, humoursome”) 
(81). Jaeger et al. (81) go on to list four conclusions that are derived from research on performance- 
standard setting for selected-response tests (but may be generalizable to performance assessments): 

(1) Performance standards rarely occur naturally and therefore the boundary between a passing 
performance and one that is not is a matter of judgement. This is the reason that some have 
labeled all standards-setting as capricious (cites Glass 1978). 

(2) Performance standards are method-dependent in that decisions made based on specifications 
of levels of performance depend to a large degree on the method used to elicit judgments on 
standards. 

(3) Those who set performance standards cannot be assumed to be trustworthy judges of the 
quality of the methods they have used. 

(4) Widely used performance standard setting methods presume the existence of an underlying 
interval scale of performance on the test or assessment for which the standards are set. 
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Both Berk (1996) and Jaeger et al. (1996) recommend including a broad representation of stakeholders on 
standards-setting panels. Both articles also strongly recommend strategies for anchoring standards in 
concrete reality through use of normative data or through explicit behavioral descriptions for each level. 

The technical approach to setting performance standards employed by the NALS was responsive 
to stakeholder input to a degree. However, while input from adult literacy content experts and practitioners 
was sought in selecting the general definition of literacy and scales adopted by the NALS, it is not clear 
that stakeholder input had any significant impact on task specifications or cut scores used to place tasks and 
task performances within the five levels of the prose, document, and quantitative literacy scales. The 
NALS’ use of IRT methods in setting and validating cut scores places it very near the empirical/theoretical 
extreme of the performance standard continuum. While this has definite advantages in terms of providing 
comparable, high quality data for research, it may pose a threat to the validity (especially consequential 
validity, see Messick 1994) of the NAAL. To avoid such threats, the NAAL might enable more stakeholder 
participation in the determination of cut scores by adopting some of the steps recommended by Berk or by 
Jaeger and his colleagues. Such stakeholder involvement will be far more critical to the validity of the 
NAAL than it was for the NALS. The stakes for the 2002 NAAL will be higher as a result of changes in the 
policy environment that are moving the field of adult literacy toward a standards-based system of 
accountability. These changes and their implications for the NAAL are discussed in the section below. 

Policy/programmatic perspectives 

The need to seek new ways of broadening participation in adult literacy performance standard 
setting becomes even more evident as we move from technical considerations of methods for defining cut 
scores to the policy and programmatic implications of adult literacy performance measures. Within 
education policy circles, the term “performance standard” is most often used in conjunction with the term 
content standards and in the context of discussions of standards-based educational reform, i.e., the use of 
content and performance standards for purposes of accountability and monitoring the quality of educational 
systems. While content standards define the range of desirable knowledge and skills, performance 
standards answer the question of how much knowledge/skill is enough. In the 1990s, performance 
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standards (in the policy/programmatic sense) have become the driving force behind emerging 
accountability systems designed to improve educational outcomes. 

A previous paper on issues in standard setting for adult literacy (Stites, Foley, and Wagner 1 995) 
reviewed the language and logic of educational standards in the context of policy formulation for standards- 
based educational reform. The authors adopted definitions of various forms and functions of educational 
standards put forward by Husen and Tuijnman (1994) in their international review of systems for 
monitoring educational performance. Husen and Tuijnman (1994, 3) differentiate educational goals and 
standards and characterize goals as “usually couched in very general terms and ... not directly amenable to 
measurement.” A standard, on the other hand, “refers to a degree of excellence required for particular 
purposes, a measure of what is adequate, a socially and practically desired level of performance” (2). More 
specifically, educational standards are usually described in terms of a “desired level of content mastery and 
performance” (2). 

American educational policy discussions have revolved around definitions of content, 

performance, and to a lesser degree, opportunity-to-leam (OTL) standards. Content standards define 

“everything a student should know and be able to do” (National Center on Educational Standards and 

Testing (NCEST) 1992, 9). In other words, content standards describe the range of desirable knowledge 

and skills within a subject area. Performance standards specify “how much” students should know and be 

able to do. Thus, while content standards are primarily of use in framing a curriculum, performance 

standards establish benchmarks to shape expectations and to provide a basis for measuring learning 

outcomes and for imposing rewards and sanctions. Opportunity-to-leam (OTL) standards were proposed as 

a response to concerns over the potential inequity of raising expectations for all students without ensuring 

that all have an equal opportunity to meet higher expectations (NCEST 1992). 

In an ideal model of standards-based educational reform, these three types of educational 

standards are interconnected. The nature of the links between content, performance, and OTL standards 

was described in a report from the National Academy of Education (1993) as follows: 

... for meaningful and fair performance standards to be set, it is necessary to define the 
exact content areas to which these standards shall apply. Before performance can be 
fairly assessed, it is moreover necessary to determine whether all students have had 
adequate opportunities to learn the prescribed content (quoted in Husen and Tuijman 
1994,2) 
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The NALS approach to assessing literacy performance was not designed to operate within this model of 
standards-based educational reform. The decision not to connect the NALS to a standards-based reform 
model may have been justified by the fact that, at the time, there was no clear curricular reference point for 
a definition of content for adult literacy education. However, this lack of connection was more directly 
related to the fact that the design of the NALS was guided by a different model for policy research, one that 
had been developed for the earlier YALS (Kirsch and Jungeblut 1986) and the U.S. Department of Labor 
study of the literacy proficiencies of job seekers (Kirsch and Jungeblut 1992). The criteria for effective 
policy research guiding the design of the NALS are laid out in the following passage by Samuel Messick 
(1987): 



If large-scale educational assessments are to function effectively as policy research — that is, to 
provide empirically-grounded interpretations or understandings to inform policy judgements — a 
number of key features must be exhibited. Central among these are first, the capacity to provide 
data or measures that are commensurable across time periods and population groups, so that trends 
and group differences can be meaningfully examined; second, the capacity to provide correlational 
evidence to sustain construct interpretations; and third, provision for measuring diverse 
background and program factors to illuminate context effects and treatment or process difference 
( 1 58 — quoted in Kirsch and Jungeblut 1 992, 5). 

In this passage, Messick identifies several key criteria for effective policy research that guided the 
development of the NALS profile approach. Mosenthal (1997) summarizes these criteria as comparability, 
relevance, and interpretability and considers them to be the keys to optimizing links among research, 
policy, and practice. Mosenthal argues that the NALS represents an “agenda-analytic” approach that 
embodies these three criteria. 

The purpose of NALS was not to solve the ‘adult literacy problem,’ but rather to provide 
a framework for informing the national adult literacy agenda so that, within the broader 
context of the survey’s agenda, problems and goals could be identified and addressed as 
different groups of adult literacy researchers, policymakers, and practitioners saw fit. 

(292). 



The push for the development of national goals and standards for American education has created 
a difficult climate for the “agenda-analytic” approach to function as intended. In Mosenthal’s vision, the 
NALS construct-driven “profile” approach would supply the substance for the definition of an adult 
literacy policy agenda. In actuality, political processes of standards development have overtaken the 
empirical/theoretical standards-setting process of the NALS with the result that the NALS literacy construct 
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scales and levels have become just one among a number of competing, policy-sanctioned models for adult 
literacy content/performance standards. 

The rush to identify educational content and performance standards emerged out of concerns over 

the quality of the American workforce and perceptions of mediocre educational achievement by American 

schoolchildren in the mid-1980s. The standards-based education reform movement rapidly picked up speed 

and power in the late 1980s, culminating in the 1989 formulation of the America 2000 educational reform 

agenda proposed by the nation’s Governors and President Bush. America 2000 defined six National 

Education Goals, including Goal 5: Adult Literacy and Lifelong Learning: 

By the year 2000, every adult American will be literate and will possess the knowledge 
and skills necessary to compete in a global economy and exercise the rights and 
responsibilities of citizenship. 

The goals and objectives of the America 2000 proposal were adopted by the U.S. Department of Education 
in 1992 and were subsequently included in a legislative initiative entitled the “Goals 2000: Educate 
America Act.” By the time that the Goals 2000 legislation was submitted to Congress in 1993, the number 
of goals had been expanded from six to eight and the objectives were elaborated to include a set of sixteen 
core indicators. 

The 1994 National Goals Report prepared by the NEGP explains that the sixteen core indicators 
were designed to be “comprehensive across the Goals; most critical in determining whether the Goals are 
actually achieved; policy-actionable; and updated at frequent intervals, so that the Panel can provide regular 
progress reports” (NEGP 1994, 15). Furthermore, these core indicators were meant to provide general 
criteria for policymakers, educators, and the public to employ in measuring progress in raising the level of 
the nation’s “educational health”; to give policymakers and the public a better idea of what they can do to 
improve educational performance; to clearly communicate “benchmarks” for expected levels of 
performance; and to identify and remove gaps in national and state level data that might get in the way of 
the Goal Panel’s task of measuring progress toward the National Goals (14-15). Among the three 
indicators that were specified for Goal 6 (originally Goal 5): Adult Literacy and Lifelong Learning in the 
1994 Goals Report, the first is of particular interest here. 

Indicator 10 — Adult literacy: 

Increase the percentage of adults aged 16 and over who score at or above Level 3 in prose 
literacy on the National Adult Literacy Survey (NALS) (NEGP 1 994, 41 ). 
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The NEGP’s choice of a level of performance on one of the three scales developed for the NALS 
as a core indicator of educational progress thus forced the NALS “agenda-analytic” intent into the 
Procrustean bed of a standards-based model of educational reform and accountability. Designed as an 
agenda setting profile of the distribution of adult literacy proficiencies, the NALS became instead a 
barometer of the nation’s “literacy health” and a “benchmark” of expected literacy performance. The 
NALS literacy scales and levels filled an apparent gap left by the absence of any complete and coherent 
system of content and performance standards for adult literacy. It remains to be seen whether such a system 
will be in place by the time the NAAL is conducted in 2002. Currently, a number of policy initiatives 
(some begun before the release of the NALS report and some after) are moving in the direction of national 
content and/or performance standards for adult literacy. 

Since the mid-1980s there has been a proliferation of educational standards-setting activity in the 
U.S. at both the national and the state levels. At the national level, the leader and pacesetter in these efforts 
has been the National Council of Teachers of Mathematics (NCTM). One of the central lessons ofNCTM’s 
decade-long experience (see Ball 1992) and of the experiences of other K-12 subject area standards-setting 
efforts in attempting to follow their lead has been that moving from the theory of standards and standards 
policy to the actual process of setting standards has required customization of the forms and functions of 
standards to suit the particular needs and characteristics of each subject area. Not surprisingly, given the 
different institutional contexts for K-12 subject area standards and adult literacy standards, the setting of 
standards for adult literacy seems to be pursuing its own unique path. In view of the fragmentary, 
“patchwork” quality of the adult basic education system (see Office of Technology Assessment 1993; 
Young et al. 1994), it is not surprising that standard setting for adult literacy has also been a fragmented 
and disparate enterprise. Current (and in some ways competing) models for national content/performance 
standards for adult literacy include but are not limited to the foundation skills defined by the U.S. 
Department of Labor for the SCANS and the occupational skills standards developed under the auspices of 
the NS SB, the NIFL’s Equipped for the Future framework for competent adult performance, the learning 
gains measures recommended by OVAE’s National Reporting System, as well as the learning measures 
specified in state plans for adult education drafted in response to mandates in Title II of the 1998 
Workforce Investment Act. Though none of these initiatives has been focused solely on defining desirable 
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types and levels of literacy (and numeracy) skills, each has addressed literacy performance standards within 
a broader framework of skills and competencies. 

Within the realm of adult literacy standards (as in other content areas), the boundary between 
definitions of content and performance standards has tended to blur. Determinations of what one needs to 
know and be able to do to at various levels of literacy (and numeracy) proficiency lead naturally to 
considerations of how much one needs to know and be able to do to achieve a specified level of literacy 
proficiency and vice versa. However, early efforts at standards-setting for adult literacy tended to focus first 
on content standard definition. SCANS, the work of the NSSB, and NIFL’s Equipped for the Future 
initiative fall into this category. More recently, the press for learning gains measures and standards for 
program reporting and accountability has accelerated the development of performance standards work. 
OVAE’s National Reporting System and state plans for adult education written in response to Title II of the 
1998 Workforce Investment Act fall into the latter category. 

When it was formed in 1990, the SCANS was asked to “examine the demands of the workplace 
and whether [America’s] young people are capable of meeting those demands” (SCANS 1991, xv). This 
general mission was broken down into four tasks: 

-to define skills needed for employment; 

-to propose acceptable levels of proficiency in these skills; 

-to suggest effective ways of assessing levels of proficiency; and finally, 

-to develop a means of disseminating the results of their work to schools, businesses, and homes. 
The task of identifying and defining skills was carried out in a five-stage process that entailed: 
consultations with policymakers, business leaders, and a review of relevant research; the convening of 
expert panels; reviews of psychological, educational, and business databases; further consultations with 
research and business experts; and finally, analyses of skill demands of jobs in various areas of the 
economy (Whetzel 1993). The result was a model of “workplace know-how” that specified desirable 
competencies in five domains and foundation skills in three domains. Among the SCANS foundation skills 
are the following: 

Basic skills — reading, writing, arithmetic and mathematics, speaking, and 
listening (SCANS 1991, vii). 
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The generic standard for workplace literacy put forward by SCANS has been elaborated in various 
ways (O’Neil, Allred, and Baker 1992; Wills 1998), particularly in response to Title V, The National Skill 
Standards Act of the 1994 Goals 2000: Educate America Act. The Skill Standards Act established a 
National Skill Standards Board (NSSB) with the goal of ensuring “a high skills, high quality, high 
performance workforce, including the most skilled front-line workforce in the world.” To help accomplish 
this goal the NSSB initiated a “voluntary national system of skill standards and of assessment and 
certification of skill standards.” Informed by the U.S. Department of Education’s OVAE study of 
Occupational Skill Standards projects, the NSSB was charged with identifying occupational clusters as well 
as the skills and personal qualities needed to succeed in each cluster. The Skill Standards Act also enabled 
the NSSB to award grants to industry councils or other voluntary partnerships that want to develop skill 
standards and encouraged the development of a variety of voluntary certification and assessment systems 
for the skills (see Wills 1 998). 

The NIFL began its “consumer-driven” adult literacy content standards initiative in response to the 
1993 Congressional mandate to measure progress toward National Education Goal 6. The first phase of the 
initiative consisted of a survey of learners to determine what they needed to know and be able to do to 
achieve Goal 6. A content analysis of responses from 1500 respondents yielded four fundamental purposes 
for learning: access to information; “voice” or being able to express ideas and opinions with confidence; 
being able to solve problems independently; and building a bridge to the future (learning how to learn) 
(Stein 1995). The results of this survey provided the basis for the Equipped for the Future (EFF) 
framework. NIFL furthered the development of the EFF framework by awarding three planning grants in 
1996-97 for the development of “role maps” for adult performance in the roles of worker, 
citizen/community member, and parent/family member. The “areas of responsibility” and “key activities” 
developed within each of these role map development efforts were subsequently combined into a set of 
“common activities used to carry out EFF adult roles.” Over the next two to three years, NIFL hopes to 
make the Equipped for the Future standards “more explicit and measurable” and to develop a “performance 
continuum with levels and standards” so that “the standards will be used for accountability purposes.” (S. 
Stein — personal communication, May 27, 1999). 
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While the EFF standards initiative is moving gradually from content to performance standards for 
adult literacy, other initiatives are attempting to fill the immediate need for a more coherent and uniform set 
of performance standards for program accountability. 

The U.S. Department of Education’s OVAE initiated the National Outcome Reporting System 
Project (NRS) in response to a resolution passed at a 1996 meeting of state directors of adult education in 
Columbia, Maryland: 

We recommend a collaboratively funded and managed project to analyze and synthesize 
accountability systems that have been developed nationally and in separate states that 
incorporate adult education outputs and outcomes. The project will continue the next 
steps of work begun here by state directors to draft protocols, determine how data would 
be collected and how reliability could be optimized. The project will involve state 
. directors of adult education and other stakeholders in setting project policy and in project 
operations (quoted in Condelli and Kutner 1997, 1). 

Among the seven categories of outcome measures developed by state directors at the 1996 meeting was the 
following: 

Learning gains — measures that demonstrate that the participant acquired reading, writing, 
functional or employment-related skills, numeracy, or English-speaking and listening 
skills (2). 

In their 1997 report on the development of the NRS, Condelli and Kutner (1997) make the 

following comments on the issue of defining performance standards: 

Adult education stakeholders are still clarifying the policy goals, measures and methods 
for the outcome reporting system and, therefore, may not want to set explicit performance 
standards at this time. However, the issue cannot be ignored because, at some time, key 
audiences will demand a performance standard, or impose their own on the program (i.e., 
they will judge the program according to their own criteria) (60). 

A 1998 report on definitions of measures for the NRS (Condelli 1998) notes that two factors 
guided the development of the NRS definitions: the need to accommodate the diversity of the adult 
education delivery system and the need for compatibility of the definitions with related adult education and 
training programs. The section of the report on learning gains notes that stakeholders have indicated that 
“the adult education program is primarily an educational program designed to teach literacy skills 
[emphasis in the original]” (Condelli 1998, 8). The report includes “entry level descriptors of student 
functioning in content areas” (basic reading and writing; numeracy skills; and, functional and workplace 
skills) as well as “test benchmarks” equating six “literacy levels” for adult basic education to Test of Adult 
Basic Education (TABE) scale scores in reading and math and Comprehensive Adult Student Assessment 
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System (CASAS) scores, and four “literacy levels” for English as a second language corresponding to 
scores on the CASAS and Student Performance Levels (SPL) speaking and reading levels. 

The most recent source of increased pressure for the development of performance standards for 
adult literacy program accountability came with the 1998 passage of the federal Workforce Investment Act 
(WIA). Title II of the WIA, the Adult Education and Family Literacy Act, replaces the former Adult 
Education Act. The WIA places greater emphasis on documentation of learning gains than on numbers of 
learners participating in an adult education program. It does so by tying receipt of federal funding by states 
to the development and implementation of five-year plans for improving instructional and professional 
development outcomes. At least 33 states have drafted state plans for adult education in response to 
mandates for such plans in the WIA. Several states have made draft versions of their plans available on the 
Internet, including California, Illinois, Iowa, Tennessee, and Washington. The California State Plan uses 
NALS levels from California’s State Adult Literacy Survey (SALS) data as markers for establishing needs 
for adult basic education and for setting priorities for populations to be served. However, learner outcomes 
in the California State Plan are specified in terms of CASAS and TABE scores. Moving in parallel and in 
some ways ahead of national trends, some states, notably Iowa, Pennsylvania, Kentucky, and 
Massachusetts, have made headway in creating coherent systems for reporting adult basic education 
outcomes at the state level. For performance measures, states have tended to take the pragmatic route of 
relying on commonly used standardized tests (e.g., CASAS and TABE) as recommended by the OVAE’s 
National Reporting System. 

To better grasp the ways that the various content and performance standards initiatives described 
above are or are not filling the need for policy/programmatic adult literacy performance standards and to 
understand the role that these developments suggest for the NAAL vis-a-vis emerging accountability 
frameworks, it will be useful to look more closely at the range of functions for performance standards 
within accountability systems. Also, given the pace and volume (if not coherence) of adult literacy content 
and performance standards setting, the field of adult literacy may be moving into a status roughly parallel 
to K-12 subject areas where national standards are layered over diverse local curricula and performance 
measures. In such a case, the lessons of the NAEP may become relatively more relevant to the design of the 
NAAL. 
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Eva Baker (1996) adopted an international comparative perspective to analyze the functions of 



performance standards in an educational accountability system. She found that in the context of U.S. 
educational reform, performance standards have been defined as means of strengthening “systems 
processes of control, coherence, guidance, and participation, as well as to set clearer boundaries for the 
measurement quality of educational attainments” (6). The five functional dimensions of performance 
standards identified by Baker (1996) provide a useful organizing framework for considering the potential 
roles of adult literacy performance standards. 

• Control — performance standards are useful to the extent that they “transfer attention 
away from specification of educational process” and yet the manner in which 
performance standards are phrased may communicate preferences for particular 
educational methods (Baker 1996, 10). 

NALS performance standard setting processes and reporting seem to have had only a very indirect 
effect on shifting the focus of the adult basic education system away from educational process concerns and 
toward a focus on educational attainments. Control functions for the next NAAL performance standards are 
likely to operate through the correspondence (or influence) of the NAAL multidimensional construct of 
literacy proficiencies on definitions of content standards (EFF, NSSB skills standards) and accountability 
mechanisms (NRS and state plans). In the latter case, the linking of NAAL and CASAS performance levels 
(and performance level descriptors) may be attempted as a means of connecting NAAL-based demographic 
profiles to CASAS-based program outcomes. Such linking is not advisable. Feuer and colleagues (1997, 
see also Mislevy 1992) have pointed out multiple factors (including content, format, margins of error, 
intended and actual uses, and consequences of the tests) that affect the validity of inferences drawn from 
linked scores. 

• Coherence — performance standards may be a source of coherence (the “extent to 



which students in an educational system share in common experiences that would 
provide policymakers and teachers with a clear understanding of the order and nature 
of their learning”) in the U.S. decentralized educational system where other potential 
influences on coherence (curriculum, materials, teacher training) are indirect and 
weak (Baker 1996, 1 1). 



The perceived mismatch between the construct of literacy skills guiding the NALS and the 
learning objectives of most adult basic education curricula raises questions about the potential contribution 
of the NAAL to coherence in the adult education system. One of the perceived strengths of performance 
standards and the alternative forms of assessment that they support is the potential that they hold to clearly 
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communicate expectations for student achievement and at the same time to more closely link classroom 
instruction and assessment to accountability measures (see Herman, Aschbacher, and Winters 1992). As a 
profile of the distribution of literacy proficiencies in the adult population, the NALS had virtually no 
influence on the coherence of the adult basic educational system. In some ways, the influence of the NALS 
and the potential influence of the NAAL on coherence in the adult literacy field is mitigated by the fact that 
the field has long been oriented toward the functional literacy perspective that guided the design of the 
NALS and related literacy assessments. The orientation toward a functional literacy perspective was largely 
the result of the field’s earlier exposure to the Adult Performance Levels work in the 1 970s (James 
Parker — interview, April 7, 1999). Even so, the potential impact of the NAAL on adult literacy curricular 
and assessment practices would be increased if NAAL task content were better aligned with adult literacy 
curricula and learning goals. One way to achieve such alignment may be to incorporate the purposes, roles, 
and common activities of the EFF framework in the design of new tasks for the NAAL. This would allow 
for some alignment of the NAAL to emerging adult literacy content standards, curriculum, materials, and 
teacher training. 

♦ Guidance — if they are sufficiently detailed, performance standards may provide 

guidance (“forms and types of information provided that are needed to generate 
willing compliance by participants”) to an educational system by 1) clarifying “the 
order and nature of expectations for different-aged learners or for same-aged students 
at different attainment levels,” and 2) providing cues (“essential quality criteria”) for 
the organization of instruction (Baker 1996, 12-13). 

It is clear that the NALS did not provide the forms of guidance detailed above, nor was it designed 
to do so. The lack of alignment of the NALS with adult literacy curricula and learning goals was the chief 
limiting factor. As noted above, such alignment could be enhanced for the NAAL and NAAL performance 
standards (in the form of performance level descriptors) may provide guidance to educators and adult 
learners if they are sufficiently detailed and disseminated in appropriate ways. The Internet and multimedia 
formats could be used to disseminate descriptions of the NAAL tasks, scales, and level descriptors. 
However, simply popularizing the details of the NAAL literacy constructs and scales may not suffice to 
convince adult educators that these are appropriate or important guides for literacy learning and instruction. 
Broad-based consensus on detailed definitions of literacy has proven hard to achieve (see Venezky, 

Wagner, and Ciliberti 1 990 and the popular perspectives section below). 
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• Participation — patterns and degrees of constituency participation in setting 
performance standards are influenced by 1) “traditions of responsibility and 
authority, general satisfaction with the system’s effectiveness and scope, and the 
diversity of the publics served by educational programs,” and 2) “the technical 
character of the approach taken” (Baker, 1996 14). 

As noted in the technical perspectives section above, the question of who participates in the setting 
of performance standards for the next NAAL must be carefully weighed and a more broad-based 
participatory process may be possible without seriously threatening the reliability and validity of the 
process. 

• Measurement quality — Baker describes three dimensions of the development and use 
of educational outcome measures: credibility, quality, and adaptability. Credibility 
issues have arisen because of the tight timetable for educational change. Performance 
standards suggest more performance-based assessment but “skepticism about the 
difficulty, fairness, and trustworthiness of new examinations” has undermined some 
educational reform agendas. High-quality measures create credibility. Again time is 

a factor, but will sufficient time and resources be available “to develop the 
appropriate scientific base for new assessments before skepticism, overpromising, 
and retreats to earlier measurements approaches take over?” Finally, adaptability is 
“the system’s capacity to expand, contract, or change direction over time” yet, in the 
U.S., system change “is less a feature of macroplanning than the happenstance 
confluence of resources, politics, and innovative ideas” (Baker, 1996, 15-17). 

Baker’s concerns about the impatience of accountability systems seem to be very much on target 
in the case of developing and selecting appropriate measures for adult literacy learning outcomes (see 
above). The credibility, quality, and adaptability of the NAAL measures will be tested severely by the 
policy environment and expectations that are likely to await the release of the 2002 results. If the NAAL 
takes up the challenge of performance standards-based reporting linked to adult literacy education goals 
and content, it may encounter problems similar to those faced by the NAEP in reporting “achievement 
levels.” The first use of performance standards-based reporting for the NAEP results were the 
“achievement levels” developed by the National Assessment Governing Board (NAGB) for the 1990 
mathematics assessment. NAGB interpreted its establishing legislation as giving it a mandate to set 
performance standards for NAEP. Baker and Linn (1997, 20) summarize the results of three separate 
evaluations of the NAGB’s initial effort to set achievement levels for the 1990 mathematics assessment 
(Linn, Baker, and Dunbar 1991; Stufflebeam, Jaegar, and Scriven 1991; U.S. General Accounting Office 
1993). These evaluations included the following criticisms: 

• The achievement level descriptions and associated exemplar items did not adequately 
coincide with actual performance of students scoring at a given achievement level; 
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• There was a lack of evidence to support the validity interpretations invited by the achievement 
level descriptions; 

• The NAEP item pool was not adequate to measure advanced levels; 

• The judgement process was too demanding for raters; and 

• The standards were overly dependent on the particular sample of judges. 

Although the NAGB responded to these early criticisms, Baker and Linn (1997, 21) point out that both the 

1992 mathematics and reading achievement levels were judged to be unacceptable in two other evaluations 

(National Academy of Education 1993; U.S. General Accounting Office 1993). 

Impatience was evident in the National Education Goals Panel's (1994) use of the NALS 

(“increase the percentage of adults age 1 6 and older who score at or above Level 3 in prose literacy on the 

National Adult Literacy Survey” (41)) as an indicator for the adult literacy component of Goal 6. This 

choice sent the message (whether intended or not) that Level 3 on the NALS prose literacy scale was the 

benchmark (threshold level) for functional literacy. The impression of Level 3 as a benchmark for 

functional literacy is reinforced by the following explanation of this choice in the 1994 report: 

Although adults who score below Level 3 do have some limited literacy skills, they are 
not likely to be able to perform the range of complex literacy tasks that the National 
Education Goals Panel considers important for competing successfully in a global 
economy and exercising fully the rights and responsibilities of citizenship. 

While Level 3 on the prose scale may well represent an important minimal threshold of proficiency in some 

respects — the GED-NALS comparison study (Baldwin et al. 1 995), for example, provides some justification 

for the choice in showing the correspondence of Level 3 performance on the NALS scales to passing 

performance on the GED Tests — the choice of a single scale and the implied dichotomization of adults into 

functionally literate (at or above Level 3) and functionally illiterate (Levels 1 or 2) is unfortunate and 

misleading. As discussed in the section on popular perspectives below, leading the public away from such 

dichotomous thinking about functional literacy and illiteracy will pose significant challenges for the 

NAAL. 
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Popular perspectives 



The general public and the news media seem to have fairly fuzzy understandings of performance 

standards. In the case of adult literacy performance standards, the “folk” concept seems to be linked to 

notions of being able to perform everyday reading and writing tasks, especially those encountered in the 

workplace. The NALS literacy definition and scales seem to have been widely misunderstood by lay 

audiences. One bit of evidence on the depth of popular misunderstanding of the NALS can be found in 

Education Week's September 15, 1993 story on the release of the NALS report. Under the headline “Half 

of adults lack skills, literacy study finds,” the first sentence of the story is as follows: 

Nearly half of all adult Americans cannot read, write, and calculate well enough to 
function fully in today’s society, and people in their early 20’s have poorer literacy skills 
than did those in a 1985 survey, according to a federal study. 

This summary statement of the NALS findings contains several rather serious and probably common 

misconceptions. First, the statement that half of all adult American lack adequate writing skills is an 

apparent misunderstanding of the nature of the NALS document literacy measures. Second, the association 

of performance at NALS Levels 1 and 2 with functional illiteracy is never made in the NALS report. In 

fact, this association is directly contradicted by a bulleted point in the report’s Executive Summary (Kirsch, 

et al. 1993) that reads (in part): 

The approximately 90 million adults who performed in Levels 1 and 2 did not necessarily 
perceive themselves as being ‘at risk.’ ... It is therefore possible that their skills, while 
limited, allow them to meet some or most of their personal and occupational literacy 
needs (xv). 

Third, the NALS report attributes its finding of relatively lower performance by adults in their 20s to 
changing demographics and particularly to “the dramatic increase in the percentages of young Hispanic 
adults, many of whom were bom in other countries and are learning English as a second language” (xvi). It 
is certainly not fair to judge the quality of public (mis)understanding of the NALS report by one newspaper 
article. As a matter of fact, the author of the Education Week article goes on to provide a short but accurate 
description of the NALS items and methods of scaling items for levels of difficulty and also notes the 
impact of immigration on the relative performance of the young adults in the 1985 and 1992 surveys. In 
other words, this reporter seems to have read the NALS report carefully and her summary of its findings is 
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relatively nuanced and accurate. Nonetheless, a reader of the Education Week report on the NALS would 



not be wrong in drawing the conclusion that adults who performed below Level 3 on the NALS prose, 

document, and quantitative literacy scales did not meet the standard for functional literacy. 

There seems to be little doubt that the results of the NAAL may be similarly misinterpreted by 

many policymakers, by the general public, as well as by many within the adult literacy educational system. 

The question is how to present the results of the NAAL in a way that avoids inappropriate uses of the 

results in evaluating adult literacy performance and, even more importantly, avoiding inappropriate 

applications of the NAAL results to adult educational program accountability. As noted in the above 

policy/programmatic section above, one positive step might be to link the NAAL results to a more specific 

and detailed definition of literacy than that adopted by the NALS. 

The NALS was guided by a definition of literacy originally developed by a national panel of 

experts for the YALS assessment (Kirsch and Jungeblut 1986) as follows: “Using printed and written 

information to function in society, to achieve one’s goals, and to develop one’s knowledge and potential” 

(Kirsch et al. 1993, 2). According to Campbell, Kirsch, and Kolstad (1992, 9-10), the expert panel 

convened by ETS to define literacy for the NALS started with this YALS assessment definition and after 

much discussion concluded that revising the definition “would narrow rather than broaden the concept of 

literacy” (10) and therefore ended where they began by unanimously adopting the YALS assessment 

definition as a guide for the NALS. The drafters of the National Literacy Act of 1991, while borrowing 

language from the YALS definition, elaborated upon that definition as follows: 

For the purposes of this Act the term ‘literacy’ means an individual’s ability to read, 
write, and speak in English, and compute and solve problems at levels of proficiency 
necessary to function on the job and in society, to achieve one’s goals, and develop one’s 
knowledge and potential 
(National Literacy Act of 1 99 1 , Section 3). 

It is a matter of perspective whether the specification of English, numeracy, and job-related 
proficiencies in the National Literacy Act wording narrows or broadens the definition of literacy. The 
wording of the National Education Goal for adult literacy is even more general than the NALS and 
National Literacy Act definitions, and the form of literacy proficiency linked to “the rights and 
responsibilities of citizenship” in the adult literacy goal is not clearly specified in either the NALS or the 
National Literacy Act definitions. Differences in emphasis aside, the basic problem with all three of these 



24 




definitions in terms of providing guidance to the public in understanding what was actually measured by 
the NALS is their high level of generality. It is only when one turns to the detailed performance level 
descriptors for the NALS scales that one can begin to appreciate the complexity of the literacy construct 
guiding the NALS measures of literacy proficiency. 

In this respect, the problems in reporting results of the NALS and the NAAL are analogous to 
those encountered by the NAGB in reporting the NAEP results. Hambleton and Slater (1997) investigated 
the extent to which NAEP Executive Summary Reports were understandable to policymakers and 
educators. They found that misunderstandings and mistakes were common in reading the NAEP report and 
attributed these to limited prior exposure to the NAEP, unfamiliarity with the NAEP reporting scale, and 
limited knowledge of statistics. As potential remedies, they recommended field testing NAEP data displays, 
simplifying NAEP reports for policymakers and educators, and tailoring NAEP reports to particular 
audiences. 

Tailoring reports of the NAAL to specific audiences may well be advisable. However, simply 
providing adult educators with more detailed understandings of the literacy constructs employed in the 
NAAL measures may not be enough. Simpler and more generic definitions of literacy have the distinct 
advantage of enabling broad-based consensus. The disadvantage of simple, generic definitions of literacy is 
that they allow too much leeway for erroneous interpretations. As noted above, when more detailed 
definitions of literacy are put forward, consensus tends to evaporate (see Venezky, Wagner, and Ciliberti 
1990). Lack of consensus on specific features of desirable literacy knowledge and skills at the level of 
learning goals and expectations is the principal obstacle to standards-based reform of the adult literacy 
educational system. The 2002 NAAL presents a rare opportunity to formulate and disseminate a 
sophisticated model of adult literacy proficiency. Taking full advantage of this opportunity will require 
coordination with other models of adult literacy emerging from the adult literacy standards and 
accountability initiatives described in the policy/programmatic perspectives section above. 
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Summary and recommendations 



Defining and reporting performance standards for the NAAL will be a complex and challenging 
enterprise. In concluding an essay on the prospects for using performance-based assessment for 
accountability purposes, Leigh Burstein 1 (1994, 9) quoted the following passage from the Underachieving 
Curriculum (McKnight et al. 1987): 

Complex enterprises generate complex problems requiring equally complex solutions. 

Schooling is such an enterprise. Therefore solutions to problems must, inevitably, be complex 
. . . The longing for simplicity in the face of essential complexity is likely to produce 
deceptive explanations that lead to ineffective solutions (51). 

Burstein followed this quotation by noting that the words “assessment” or “accountability” could be 

substituted for “schooling” in the passage without altering the central message. Setting performance 

standards for adult literacy is also a complex problem that will require equally complex solutions. 

All audiences for the NAAL will be longing for simplicity. But the complexities of adult literacy 
proficiency must be conveyed. Although, the public and policymakers will almost certainly be looking to 
the results of NAAL to answer the question of how many American adults are “literate enough,” the 
designers, reporters, and interpreters of the NAAL should resist the temptation to use the NAAL results to 
directly address this question. Many forces will tend to push in this direction. The public will expect the 
NAAL to show them where the dividing line is between functional illiteracy and functional literacy and 
what proportion of American adults fall on either side of that line. Business people will want to know what 
percentage of the labor force is prepared to meet the literacy demands of the workplace and what 
percentage is not. Government officials will want to know what impact public funding for education has 
had on adult literacy proficiency and how great the need is for further funding. Practitioners in the field of 
adult literacy education will be looking to NAAL as a source of guidance for program planning. A 
substantial public education effort will be needed to enable these stakeholders to make appropriate use of 
the NAAL results. 



1 This paper is headed by the following note from Ron Dietel, CRESST Director of Communications: 
“Leigh Burstein passed away on July 7, 1994. In honor of his memory, we are publishing this report with 
virtually no editorial changes. We ask anyone who references or quotes from this paper to note that Leigh 
did not review the final publication. It is possible that Leigh would have made significant changes or none 
at all.” 
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Defining “appropriate” interpretations and uses of NAAL results will entail a combination of 
political (policy) and technical (measurement) decisions. Decisions in the measurement arena (i.e., task 
content and choice of methods to use in defining cut scores to place tasks and individuals within specified 
levels of the literacy performance continua) will shape the context for decision making in the policy arena 
(i.e., public and policymaker’s perceptions of the need for and value of adult basic education). This paper 
has examined the intersection of these two decision making processes. It has looked back at the experience 
of the NALS and forward to the lessons of that experience — in light of new technical approaches to 
performance standard setting and a changed climate of adult literacy standards policy and accountability 
systems — for performance standard setting and reporting for the next National Assessment of Adult 
Literacy. 

The key challenges facing the 2002 NAAL in defining and reporting performance standards have 
arisen from the following conditions: 

• The NALS “profile” approach to large-scale literacy assessment was not intended to 
set a performance standard for adult literacy (in the sense of answering the question 
of “how much literacy is enough”). Nonetheless, the NEGP’s decision to adopt Level 
3 on the NALS prose scale as an indicator of progress toward the National Education 
Goal for adult literacy created a de facto NALS-based national performance standard 
for adult literacy. 

• The “agenda-analytic” criteria for effective policy research that guided the design of 
the NALS are at odds with the assessment-driven, standards-based model of 
educational reform that has come to dominate American educational policy 
discourse. 

Recent developments in measurement methodology and in educational standards policy 
have created new opportunities and challenges for the NAAL. 

• Psychometricians have developed methods for including stakeholders in the 
definition of performance standards for large-scale assessment. 

• Policy initiatives have moved the field of adult literacy education toward a 
standards-based model of system reform and accountability. 
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Because performance standards mean different things in different contexts, the discussion of the 
above challenges was framed by a continuum of conceptualizations of adult literacy performance standards. 

• At one end of the continuum are “technical” conceptions of performance standards 
as used in the psychometric literature in discussions of methods for setting cut scores 
for levels of proficiency in large-scale assessments. 

• At the opposite end of the continuum are “popular” conceptions of adult literacy 
performance standards as expressed in everyday discourse and in the news media. 

In between these two extremes are “policy/programmatic” conceptions of adult literacy 
performance standards defined in educational goals (NEGP, SCANS, NS SB, NIFL’s 
Equipped for the Future), accountability systems (NRS, WIA state plans), as well as in adult 
literacy program curricula and learning goals. 

There are numerous gaps separating technical, policy/programmatic, and popular conceptions of 
adult literacy performance standards, but some of these gaps can and should be bridged. The NAAL 
presents opportunities for bridging gaps in the following ways: 

Recommendations for technical performance standards: 

• Adopt methods for including a broad-range of stakeholders in the process of defining 
technical performance standards (task specifications and cut scores) for the NAAL. 

Recommendations for policy /programmatic performance standards: 

• Emphasize applications of NAAL results to profile populations in need of literacy 
education services. 

• Avoid misapplications of the NAAL results for adult education program 
accountability. 

• Contrast literacy constructs and skills measured by the NAAL and those measured 
by standardized tests (CASAS, TABE, etc.) used in accountability systems in order 
to avoid linking or equivalencies of the NAAL scales and levels to program outcome 
measures. 
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• Make connections between NAAL literacy definitions and performance levels and 
adult literacy program curricular content and content standards (such as EFF) so that 
the literacy constructs informing the NAAL scales and levels become more 
transparent and can serve as guides for program planning and goal-setting. 

Recommendations for popular performance standards: 

• Conduct public education campaigns to provide concrete and clear explanations of 
the NAAL scales and level descriptors and to avoid popular misinterpretations of 
NAAL performance Levels 1-2 as “functional illiteracy.” 
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National Longitudinal Study of the High School Class of 1972 (NLS-72) 

95-12 Rural Education Data User’s Guide 
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National Postsecondary Student Aid Study (NPSAS) 

96- 1 7 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 

National Study of Postsecondary Faculty (NSOPF) 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

98- 1 5 Development of a Prototype System for Accessing Linked NCES Data 

2000-0 1 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 



Andrew G. Malizio 



Linda Zimbler 
Steven Kaufman 
Linda Zimbler 



Private School Universe Survey (PSS) 

95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 17 Estimates of Expenditures for Private K- 1 2 Schools 

96- 1 6 Strategies for Collecting Finance Data from Private Schools 
96-26 Improving the Coverage of Private Elementary- Secondary Schools 

96- 27 Intersurvey Consistency in NCES Private School Surveys for 1993-94 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97- 22 Collection of Private School Finance Data: Development of a Questionnaire 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 



Steven Kaufman 
Stephen Broughman 
Stephen Broughman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 

Stephen Broughman 
Steven Kaufman 
Dan Kasprzyk 



Recent College Graduates (RCG) 

98-15 Development of a Prototype System for Accessing Linked NCES Data 



Steven Kaufman 



Schools and Staffing Survey (SASS) 

94-0 1 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American 
Statistical Association 

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) 

94-03 1 99 1 Schools and Staffing Survey (SASS) Reinterview Response Variance Report 

94-04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher 

Transcript Study, Schools and Staffing Survey 

94- 06 Six Papers on Teachers from the 1 990-9 1 Schools and Staffing Survey and Other Related 

Surveys 

95- 01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American 

Statistical Association 

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing 
QED School Estimates with CCD Estimates 
95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis 
95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates 
95-09 The Results of the 1993 Teacher List Validation Study (TLVS) 

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive 



Reconciliation 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Wprk 

95-12 Rural Education Data User’s Guide 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

95-1 5 Classroom Instructional Processes: A Review of Existing Measurement Approaches and 
Their Applicability for the Teacher Follow-up Survey 
95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 1 8 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and 

Staffing Survey 

96- 01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly 

Longitudinal Study 

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting 
of the American Statistical Association 

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey 
96-06 The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to 
Inform Broad Education Policy 
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96-07 

96-09 

96-10 
96-1 1 

96-12 

96-15 

96-23 
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96- 28 

97- 01 

97-07 

97-09 

97-10 

97-11 

97-12 

97-14 

97-18 

97-22 

97-23 

97-41 

97-42 

97- 44 

98- 01 
98-02 
98-04 
98-05 

98-08 

98-12 

98-13 

98-14 

98-15 

98-16 

1999-02 

1999-04 

1999-07 

1999-08 

1999-10 

1999-12 

1999-13 

1999-14 

1999-17 



Should SASS Measure Instructional Processes and Teacher Effectiveness? 

Making Data Relevant for Policy Discussions: Redesigning the School Administrator 
Questionnaire for the 1998-99 SASS 
1998-99 Schools and Staffing Survey: Issues Related to Survey Depth 
Towards an Organizational Database on America’s Schools: A Proposal for the Future of 
SASS, with comments on School Reform, Governance, and Finance 
Predictors of Retention, Transfer, and Attrition of Special and General Education 
Teachers: Data from the 1989 Teacher Followup Survey 
Nested Structures: District-Level Data in the Schools and Staffing Survey 
Linking Student Data to SASS: Why, When, How 
National Assessments of Teacher Quality 

Measures of Inservice Professional Development: Suggested Items for the 1998-1999 
Schools and Staffing Survey 

Student Learning, Teaching Quality, and Professional Development: Theoretical 

Linkages, Current Measurement, and Recommendations for Future Data Collection 
Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the 
American Statistical Association 

The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 
Status of Data on Crime and Violence in Schools: Final Report 
Report of Cognitive Research on the Public and Private School Teacher Questionnaires 
for the Schools and Staffing Survey 1993-94 School Year 
International Comparisons of Inservice Professional Development 
Measuring School Reform: Recommendations for Future SASS Data Collection 
Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and 
Analysis 

Improving the Mail Return Rates of SASS Surveys: A Review of the Literature 
Collection of Private School Finance Data: Development of a Questionnaire 
Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing 
Form 

Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting 
of the American Statistical Association 

Improving the Measurement of Staffing Resources at the School Level: The Development 
of Recommendations for NCES for the Schools and Staffing Survey (SASS) 
Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 
State Assessments and State NAEP, Feasibility Study 
Collection of Public School Expenditure Data: Development of a Questionnaire 
Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 
Geographic Variations in Public Schools’ Costs 

SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for 

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors 
The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 
A Bootstrap Variance Estimator for Systematic PPS Sampling 
Response Variance in the 1994-95 Teacher Follow-up Survey 
Variance Estimation of Imputed Survey Data 

Development of a Prototype System for Accessing Linked NCES Data 
A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 
Measuring Teacher Qualifications 

Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest 
Results to Improve Item Construction 
What Users Say About Schools and Staffing Survey Publications 
1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use 
Codebook 

1993- 94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 
Indian Affairs (BIA) Restricted-Use Codebook 

1994- 95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 
Secondary Use of the Schools and Staffing Survey Data 
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Adult education 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

1999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education 
Statistics 

Adult literacy — see Literacy of adults 
American Indian - education 

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 

Indian Affairs (BIA) Restricted-Use Codebook 

Assessment/achievement 
95-12 Rural Education Data User’s Guide 

95-13 Assessing Students with Disabilities and Limited English Proficiency 

97- 29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97- 44 Development of a SASS 1 993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

Civic participation 

97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94- 05 Cost-of-Education Differentials Across the States 

Course-taking 

95- 12 Rural Education Data User’s Guide 
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98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 
1 999-05 Procedures Guide for Transcript Studies 
1 999-06 1 998 Revision of the Secondary School Taxonomy 

Crime 

97- 09 Status of Data on Crime and Violence in Schools: Final Report 

Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Customer service 

1 999- 1 0 What Users Say About Schools and Staffing Survey Publications 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 

1999 AAPOR Meetings 



Data quality 

97- 1 3 Improving Data Quality in NCES: Database-to-Report Process 

Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 
1999 AAPOR Meetings 



Design effects 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 

Dropout rates, high school 

95- 07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and 

NELS:88 Sophomore Cohort Dropouts 

Early childhood education 

96- 20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1 995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1 999- 01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

Employment 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 
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Faculty - higher education 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 
2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 



Finance - 
94-05 
96-19 
98-01 
1999-07 
1999-16 



elementary and secondary schools 
Cost-of-Education Differentials Across the States 
Assessment and Analysis of School -Level Expenditures 
Collection of Public School Expenditure Data: Development of a Questionnaire 
Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 



Finance - postsecondary 
97-27 Pilot Test of IPEDS Finance Survey 



Linda Zimbler 
Linda Zimbler 



William J. Fowler, Jr. 
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Stephen Broughman 
Stephen Broughman 
William J. Fowler, Jr. 



Peter Stowe 



Finance - 

95- 17 

96- 16 

97- 07 

97-22 

1999-07 



private schools 

Estimates of Expenditures for Private K-12 Schools 
Strategies for Collecting Finance Data from Private Schools 
The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 

Collection of Private School Finance Data: Development of a Questionnaire 
Collection of Resource and Expenditure Data on the Schools and Staffing Survey 



Stephen Broughman 
Stephen Broughman 
Stephen Broughman 

Stephen Broughman 
Stephen Broughman 



Geography 

98-04 Geographic Variations in Public Schools’ Costs 



William J. Fowler, Jr. 



Imputation 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and Dan Kasprzyk 

1999 AAPOR Meetings 

Inflation 

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr. 

Institution data 

2000-0 1 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler 



Instructional resources and practices 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of Sharon Bobbitt & 

Recent Work John Ralph 

1 999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test Dan Kasprzyk 

Results to Improve Item Construction 



International comparisons 

97-1 1 International Comparisons of Inservice Professional Development 
97-16 International Education Expenditure Comparability Study: Final Report, Volume I 

97-17 International Education Expenditure Comparability Study: Final Report, Volume II, 

Quantitative Analysis of Expenditure Comparability 



Libraries 

94-07 Data Comparability and Public Policy: New Interest in Public Library Data Papers 
Presented at Meetings of the American Statistical Association 
97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 



Limited English Proficiency 

95-13 Assessing Students with Disabilities and Limited English Proficiency 
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adults 

Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1992 National Adult Literacy Survey: An Overview 
1992 National Adult Literacy Survey: Sample Design 
1 992 National Adult Literacy Survey: Weighting and Population Estimates 
1992 National Adult Literacy Survey: Development of the Survey Instruments 
1992 National Adult Literacy Survey: Scaling and Proficiency Estimates 
1 992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 
Levels 

1992 National Adult Literacy Survey: Literacy Levels and the Response Probability 
Convention 

Data Sources on Lifelong Learning Available from the National Center for Education 
Statistics 

Secondary Statistical Modeling With the National Assessment of Adult Literacy: 
Implications for the Design of the Background Questionnaire 
Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 
“How Much Literacy is Enough?” Issues in Defining and Reporting Performance 
Standards for the National Assessment of Adult Literacy 



Literacy of adults - international 
97-33 Adult Literacy: An International Perspective 



Mathematics 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 

1 999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 

Results to Improve Item Construction 

Parental involvement in education 

96- 03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 

Issues 

97- 25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 



Participation rates 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 



Postsecondary education 

1 999- 1 1 Data Sources on Lifelong Learning Available from the National Center for Education 
Statistics 



Postsecondary education - persistence and attainment 
98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

1 999- 1 5 Projected Postsecondary Outcomes of 1 992 High School Graduates 
Postsecondary education - staff 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000- 0 1 1 999 National Study of Postsecondary Faculty (NSOPF :99) Field Test Report 

Private schools 

96-16 Strategies for Collecting Finance Data from Private Schools 
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97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 

97-22 Collection of Private School Finance Data: Development of a Questionnaire 

Projections of education statistics 

1 999-15 Projected Postsecondary Outcomes of 1992 High School Graduates 

Public school finance 

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 



Public schools 

97- 43 Measuring Inflation in Public School Costs 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

98-04 Geographic Variations in Public Schools’ Costs 

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 

Public schools - secondary 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Reform, educational 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 

Response rates 

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 

School districts, public 

98-07 Decennial Census School District Project Planning Report 

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 
Processing, and Editing Cycle 

School districts, public - demographics of 

96- 04 Census Mapping Project/ School District Data Book 

Schools 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

1 999- 03 Evaluation of the 1 996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

Schools - safety and discipline 

97-09 Status of Data on Crime and Violence in Schools: Final Report 

Software evaluation 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 

Variances from NCES Data Sets 

Staff 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Staff - higher education institutions 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 
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1999-03 


Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 
Processing, and Editing Cycle 


Beth Young 


Statistical methodology 




97-21 


Statistics for Policymakers or Everything You Wanted to Know About Statistics But 
Thought You Could Never Understand 


Susan Ahmed 


Students with disabilities 




95-13 


Assessing Students with Disabilities and Limited English Proficiency 


James Houser 


Survey methodology 




96-17 


National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 


Andrew G. Malizio 


97-15 


Customer Service Survey: Common Core of Data Coordinators 


Lee Hoffman 


97-35 


Design, Data Collection, Interview Administration Time, and Data Editing in the 1996 
National Household Education Survey 


Kathryn Chandler 


98-06 


National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second 
Follow-Up: Final Methodology Report 


Ralph Lee 


98-11 


Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 


Aurora D’Amico 


98-16 


A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 


Stephen Broughman 


1999-07 


Collection of Resource and Expenditure Data on the Schools and Staffing Survey 


Stephen Broughman 


1999-17 


Secondary Use of the Schools and Staffing Survey Data 


Susan Wiley 


2000-01 


1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 


Linda Zimbler 


2000-02 


Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 


Valena Plisko 


2000-04 


Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 


Dan Kasprzyk 


Teachers 


98-13 


Response Variance in the 1994-95 Teacher Follow-up Survey 


Steven Kaufman 


1999-14 


1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 


Kerry Gruber 


Teachers 


- instructional practices of 




98-08 


The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 


Dan Kasprzyk 


Teachers 


- opinions regarding safety 
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The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 


Dan Kasprzyk 


Teachers 


- performance evaluations 
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Measuring Teacher Qualifications 
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Teachers 


- qualifications of 
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Measuring Teacher Qualifications 
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Teachers 


- salaries of 




94-05 


Cost-of-Education Differentials Across the States 


William J. Fowler, Jr. 


Variance 


estimation 




2000-03 


Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 


Ralph Lee 
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Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 
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Violence 


97-09 


Status of Data on Crime and Violence in Schools: Final Report 


Lee Hoffman 
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95-12 Rural Education Data User’s Guide Samuel Peng 

1999-05 Procedures Guide for Transcript Studies Dawn Nelson 

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson 
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